Mixed reality golf simulation and training system

ABSTRACT

A mixed reality golf simulation and training system that can use, along with a user&#39;s existing standard golf equipment, a golf ball launch monitor to track the initial ball positional data, spin and acceleration, and simulate the complete ball path and location or use complete ball tracking data and displays the actual ball path and location. Mixed reality glasses allow the display of ball tracking data over the real world view and/or an immersive display of a simulated world view, depending on the user&#39;s head or view position. Golf simulation graphical views can include various options, including simulated or panoramic photographic views of a golf course, simulated graphics and data superimposed over a real world driving range view, or simple ball tracking data superimposed over a real world view at any location.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit of and is a continuationapplication of U.S. patent application Ser. No. 15/914,789, filed onMar. 7, 2018, which is incorporated by reference herein in its entiretyand claims the benefit of U.S. Provisional Patent Application No.62/468,044 filed on Mar. 7, 2017, and U.S. Provisional Patentapplication No. 62/577,551 filed on Oct. 26, 2017, each of which isincorporated by reference herein in its entirety.

FIELD

The present disclosure relates to golf simulation, and moreparticularly, to a mixed reality golf simulation and training system.

BACKGROUND

Golf is one of the most widely enjoyed sports throughout the world.However, golf enthusiasts cannot always find access to a golf coursewhen they so desire. Accordingly, golf computer games also enjoyincreasing popularity. However, golf games are limited in their degreeof realism, and therefore, in their ability to entertain the user.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example of mixed reality glasses.

FIG. 2 is a block diagram showing an example of the components of mixedreality glasses.

FIG. 3 shows an exploded view of an example of an electronic golf ball.

FIG. 4 is a block diagram showing an example of the internal functionalcomponents of the electronic ball.

FIG. 5 shows a stand-alone ball launch monitor.

FIG. 6 is a block diagram showing an example of the internal componentsof the stand-alone ball launch monitor.

FIG. 7 is a diagram showing a configuration with a ball launch monitor,regulation golf ball, ball capture net and mixed reality glasses.

FIG. 8 illustrates an example of the user's view of a regulation golfball when looking down in preparation to hit the ball through the mixedreality glasses, showing an unobstructed real world view of the ball.

FIG. 9 illustrates an example of the user's view through the mixedreality glasses when the user looks up after hitting the ball, includingthe head-up display (HUD) and 360° virtual world displaying a golfcourse.

FIG. 10 illustrates an example of the user's view through the mixedreality glasses while in the Virtual Range mode.

FIG. 11 shows an example of the user's view through the mixed realityglasses while in the Virtual Caddy mode, with the HUD displaying balltracking data superimposed over the real world view visible when theuser looks up after hitting the ball.

FIG. 12 is a drawing showing an example of the system configuration forthe Virtual Instructor mode.

FIG. 13 is a drawing showing the users view through the mixed realityglasses of a 3D virtual avatar displayed during the Virtual Instructormode.

DETAILED DESCRIPTION

Current golf simulation technology is limited to projection screens,computer monitors and hand-held displays. These options are limited intheir degree of realism, and therefore, in their ability to beentertaining or useful to the user. A mixed reality golf simulation andtraining system, on the other hand, blends the real world with a virtualoverlay when viewed with mixed reality glasses, and therefore can bemuch more entertaining and/or instructive to a user, as describedfurther below.

The present disclosure includes a mixed reality golf simulation andtraining system (“the system”) that provides greatly increased realismand usefulness for purposes of golf practice, instruction orentertainment. In at least some embodiments, the system also includesaugmented reality (AR) display capability. The term “mixed reality,” asused herein, refers to a visualization technique in which a near-eyedisplay (NED) device uses a display element that is at least partiallytransparent to overlay (superimpose) computer-generated (virtual) imageson at least a portion of the user's field of view of the real world,where the overlaid images are positionally anchored to the real-world.The term “augmented reality,” as used herein, refers to a visualizationtechnique in which an NED device uses a display element that is at leastpartially transparent to overlay (superimpose) computer-generated(virtual) images on at least a portion of the user's field of view ofthe real world, where the overlaid images are not positionally anchoredto the real-world. In the case of either mixed reality or AR display,the overlaid (“virtual”) images may be opaque, such that they completelyobscure at least a portion of the user's view of the real world, whilein other instances, the overlaid images may be partially transparent. Insome instances, the overlaid images may cover the user's entire field ofview and/or may completely obscure the user's view of the real world.

The system uses mixed reality glasses as the primary visual feedback tothe user. The user can use regulation golf clubs, golf balls andpractice mats with the system. An active golf ball launch monitor,electronic ball, tethered ball motion sensor, or any other ball motionmeasurement device, provides the tracking and flight analysis that isused by system software to visually display the shot in a virtual golfcourse or driving range. The mixed reality glasses allow a user to seesimultaneously the golf ball and a visual overlay of the golf course inrelation to the user's visual orientation. Additionally, the mixedreality glasses provide a visual overlay of the tracking and motionanalysis data over the real-world view from balls hit with or without anet. This supports virtual data display of the ball motion and trackingdata, as well as game play elements, as the user hits the ball in a realworld environment, such as on a golf course or at a driving range. Asthe user changes their visual orientation, the virtual course is updatedto show the virtual environment in the proper orientation or the virtualdata and game play elements are displayed in proper orientation over thereal world view.

The system enables a user to hit an actual golf ball in a limitedenvironment, or in a full course or driving range environment, andobtain visual feedback of what the golf ball trajectory would be on areal golf course. The visual feedback is displayed to the user thoughthe mixed reality glasses. The visual feedback, including both golf balland club data, and virtual environment display can be used as anentertainment device. The user can virtually play a round of golf, orhit multiple shots at a particular hole or practice driving in a virtualdriving range, or the user can play an actual golf course or practice atan actual driving range and use the system to virtually display the ballmotion and tracking data over the real world view. The display can alsoinclude additional elements, such as game play elements forentertainment value.

There are several advantages to the system over the existing technology.The mixed reality glasses provide a small, portable visual feedbacksystem that is not available with systems utilizing projection screensor larger computer monitors. The mixed reality glasses also provide avirtual environment that is properly oriented to the user's field ofview and orientation, which is not available with existing systems usinglarge or smaller portable displays. The mixed reality glasses furtherprovide sensor feedback that can be used to properly train the user tokeep their head still and to keep looking at the ball during the shot,which is not available with existing systems.

The system supports several alternative modes of operation, includingVirtual range, Virtual Caddy, Virtual Course, and Virtual Instructor.Each of these modes of operation supports the use of mixed realityglasses to provide visual feedback of the ball tracking data andsimulation graphics supporting the simulation.

System Architecture and Components

Referring first to FIG. 7, the mixed reality golf simulation andtraining system in at least one embodiment has two main components:mixed reality glasses 100, which the user 400 wears, and a golf balllaunch monitor (or similar motion tracking device) 200. These twocomponents 100 and 200 may be used in combination with the user's own,standard golf equipment, including a golf club 401, a regulation golfball 403 and a golf ball shot distance limiting device 402 such as anet.

The mixed reality golf simulation and training system includes mixedreality glasses 100, also called a headset, head-mounted display (HMO)device or NED device, which include the display elements main userinterface and graphics processor. FIG. 1 shows an example of the mixedreality glasses 100, the functional components of which are shown in theblock diagram of FIG. 2. The mixed reality glasses 100 include an NEDdisplay subsystem that includes a left eye NED element 101 and a righteye NED element 102 that are either fully or semitransparent, or fullyopaque depending on the software application (i.e., they may providevariable transparency, such as through selective polarization of light).Software for performing operations of the mixed reality glasses 100 canreside in a memory 115, but may alternatively or additionally reside inan external processing system that communicates with the mixed realityglasses 100 through a wired communication interface 108 or wirelesscommunication interface 110, 111, 112. In some embodiments, the maingraphics and application processor 103 may include on-chip memory (notshown) that stores instructions and/or data for carrying out at leastsome of the operations of the mixed reality glasses 100.

In at least one embodiment, the main graphics and application processor103 controls and coordinates the overall operation of the system,including coordination of the operations of other components of themixed reality glasses 100, processing of data received from othercomponents, and generation and/or pre-processing of data for display bythe display elements 101, 102. The main graphics and applicationprocessor 103 can be or include, for example: one or more programmablemicroprocessors, microcontrollers, or the like: hardwired logiccircuitry such as one or more application-specific integrated circuits(ASICs), programmable logic devices (PLDs), field programmable gatearrays (FPGAs), or the like; or any combination of programmabledevice(s) and/or hardwired logic device(s). Memory 115 may be or includeone or more physical memory devices, each of which can be a type ofrandom access memory (RAM), such as dynamic RAM (DRAM) or static RAM(SRAM), read-only memory (ROM), which may be programmable, such as flashmemory, or any combination thereof. The term “software,” as used herein,is intended to include any sequence of instructions stored in anon-transitory machine-readable medium and for execution by aprogrammable processor, including firmware.

The mixed reality glasses 100 in the illustrated embodiment also includeone or more sensors for position detection in three-dimensional space,including an inertial measurement unit (IMU) 104 for sensing angularand/or linear rate of change and/or magnetic orientation. Additionalsensors may include an integrated camera 105 for photometric positionaldetection and Global Positioning System (GPS) sensors 106 for satellitedetection of position of the mixed reality glasses 100 relative to theearth.

The mixed reality glasses 100 may also include an audio output 107component, which may include an audio amplifier and speaker orearphones, to provide audio feedback to the user, which may includevoice narration or commands, or sound effects to provide audio cues orsound effects to further enhance the user experience. The mixed realityglasses may include a microphone 109 that allows the user to providevoice commands to the application and may monitor external sounds, suchas the ball being hit, to provide additional feedback to the softwareapplication.

The mixed reality glasses 100 may communicate with the ball launchmonitor 200, an electronic ball 300 (FIG. 3) and/or an externalprocessor using one or more wireless communication interfaces 110, 111.These interfaces may include, for example, a Wi-Fi transceiver 110, or aBluetooth transceiver 111 (e.g., Bluetooth 4.0 or 5.0). The mixedreality glasses 100 may additionally or alternatively include a customalternate wireless interface 112 that utilizes public wirelessfrequencies, such as 900 MHz that allow for spread spectrum frequencyhopping digital interface to provide a longer range (e.g., up to 20miles) wireless link that is more immune to wireless interference, forcommunication to other glasses or a main data station.

The mixed reality glasses 100 may include a user interface 113 that mayinclude a plurality of physical buttons, sliders, toggle switches ortouch sensors. This interface will allow the user to startup,initialize, setup and operate the basic functions of the glasses and/orthe user software application. The mixed reality glasses 100 may alsoinclude a user interface based on spoken commands input via themicrophone 109.

The mixed reality glasses 100 may include a battery and battery chargingcircuit 114. The batteries may be one of several current batterytechnologies including rechargeable lithium ion or nickel-cadmiumbatteries or replaceable alkaline batteries. The battery chargingcircuit can connect to an external charging voltage source using a wiredconnection or a wireless charging pad.

As mentioned above, the system can include a stand-alone ball launchmonitor 200 (FIG. 7) to track the ball's position, speed and otherpertinent data to provide an accurate representation and simulation ofthe ball flight and landing profile. The user's mixed reality glasses100 display in real-time (i.e., immediately after the ball is hit) theball position, speed and other pertinent data and provide an accurategraphical representation and simulation of the ball flight and landingprofile. An example of a launch monitor is shown in FIGS. 5 and 6. Todaythere are several manufacturers of off-the-shelf ball launch monitorsthat use different technologies to track and provide data parameters,including companies such as FlightScope, Trackman, Foresight Sports andErnest Sports, etc.

The launch monitor 200 includes several functional components, includinga main application processor 202 which, in at least some embodiments,can execute a software application. The software may be stored in memory213 coupled to or located in the application processor 202. The launchmonitor 200 may also include a display 201 and a user interface 211 forproviding a way for the user to set up, operate and view data from thelaunch monitor 200. The display 201 may be, for example, a monochrome orcolor liquid crystal display (LCD) or organic light emitting diode(OLEO) type display, with or without a touchscreen input capability. Theuser interface 211 may include various buttons, sliders, toggle switchesor touch panels to allow the user to select functions and control thesoftware application settings and operations.

In some embodiments, the application processor 202 may include on-chipmemory (not shown) that stores instructions and/or data for carrying outat least some of the operations of the launch monitor 200. In at leastone embodiment, the application processor 202 controls and coordinatesthe overall operation of the launch monitor 200, including coordinationof the operations of other components of the application processor 202,processing of data received from other components, and generation and/orpre-processing of data for display and/or for transmission to the mixedreality glasses 100. The application processor 202 can be or include,for example: one or more programmable microprocessors, microcontrollers,or the like: hardwired logic circuitry such as one or moreapplication-specific integrated circuits (ASICs), programmable logicdevices (PLDs), field programmable gate arrays (FPGAs), or the like; orany combination of programmable device(s) and/or hardwired logicdevice(s). Memory 213 may be or include one or more physical memorydevices, each of which can be a type of random access memory (RAM), suchas dynamic RAM (DRAM) or static RAM (SRAM), read-only memory (ROM),which may be programmable, such as flash memory, or any combinationthereof.

The launch monitor may also include one or more electromagnetic sensorsto determine the ball's location, spin and acceleration, and/or the clublocation, orientation and acceleration, all of which is processed by thesoftware application to determine the data associated with the predictedball flight and landing profile. These sensors may include a Dopplerradar subsystem 203 for use to determine the ball and/or club positionand speed, a camera 204 for use as a photometric measurement device todetermine the ball and/or club location, speed, spin and various otherdata that can be determined from a photometric measurement, or LightDetection and Ranging (LiDAR) subsystem 206 for highly accurate laserrange finding to determine with greater precision the ball and/or clublocation, speed, spin and acceleration. The sensors may include a GPS205 satellite receiver used to determine the ball launch monitorposition relative to the earth. The sensors may also include a tetheredball sensor 207 that uses a ball attached to a string or other tetherwhich is attached to a base allowing the ball to be hit without a net.The ball tether is attached to a sensor, or the ball passes by a sensorin the base that measures the ball location, speed, spin andacceleration. The sensors may also include a level sensor 208 used toensure that the launch monitor is properly and/or automatically leveledrelative to the ground, which further ensures other sensors' accuracyand repeatability.

The launch monitor may communicate with the mixed reality glasses 100 byone or more wireless communication interfaces 209, 210. These interfacesmay include, for example, a Wi-Fi transceiver 209 or a Bluetoothtransceiver 210 implementing one of the recent versions of Bluetooth(e.g., Bluetooth 4.0 or 5.0).

The launch monitor 200 may also include a battery and battery chargingcircuit 212. The batteries may be one of several current batterytechnologies, including rechargeable lithium ion or nickel-cadmiumbatteries or replaceable alkaline batteries. The battery chargingcircuit can connect to an external charging voltage source using a wiredconnection or a wireless charging pad. Alternatively, the launch monitor200 may include only a wired power connection to an external powersource such as a low voltage DC power converter or external batterypack.

As an alternative to a stand-alone ball launch monitor 200 (FIG. 7), thesystem can instead utilize an “electronic ball,” i.e., a ball that isessentially standard in materials and construction except that itincludes internal circuitry that internally tracks the ball's position,spin, acceleration and other data, and transmits that information to theuser's mixed reality glasses 100 in real-time via a wireless link.Hence, an electronic ball serves as its own launch monitor. The user'smixed reality glasses 100 display in real-time (i.e., immediately afterthe ball is hit) the ball position, speed and other pertinent data andprovides an accurate graphical representation and simulation of the ballflight and landing profile.

An example of such an electronic ball is shown in FIGS. 3 and 4. Theelectronic ball 300 includes several functional components, including amain application processor 301 that executes the software application toconvert raw sensor data into ball position, speed, spin and otherpertinent data. The software may be stored in a memory (not shown)located in or coupled to the application processor 301. In at least oneembodiment, the application processor 301 controls and coordinatesoperation of the functional elements within the ball 300. Theapplication processor 301 can be or include, for example: one or moreprogrammable microprocessors, microcontrollers, or the like: hardwiredlogic circuitry such as one or more application-specific integratedcircuits (ASICs), programmable logic devices (PLDs), field programmablegate arrays (FPGAs), or the like; or any combination of programmabledevice(s) and/or hardwired logic device(s).

The electronic ball 300 may also include one or more electromagneticsensors used to determine the ball location, spin and acceleration, andother data, all of which is processed by the software application topredict the ball flight and landing profile. These sensors may include,for example, an IMU 302 that measures ball linear and/or angularacceleration, magnetic orientation and other inertial measurements. Thesensors may also include a GPS receiver 303 used to determine the ball'sposition relative to the earth.

The electronic ball 300 may communicate with the user's mixed realityglasses 100 using one or more wireless communication interfaces. Thesemay include a Wi-Fi transceiver 304 or a Bluetooth transceiver 305implementing one of the recent versions of Bluetooth (e.g., Bluetooth4.0 or 5.0).

The electronic ball 300 may include a battery and battery chargingcircuit 306. The batteries may be one of several current batterytechnologies including rechargeable lithium ion or nickel-cadmiumbatteries. The battery charging circuit can connect to an externalcharging voltage source using a wireless charging pad or ball holder.

System Operation

The system transmits multiple ball position and tracking data from alaunch monitor to the mixed reality glasses 100. This data includesparameters that are captured by a stand-alone ball launch monitor 200 orinternally by an electronic ball 300. The raw captured data parametersare processed in the ball launch monitor or in the electronic ball intousable parameters that can be processed by the mixed reality glasses100. These parameters can be transmitted to the mixed reality glasses100 using one or more standard wireless communication technologies, suchas Wi-Fi or Bluetooth.

With a stand-alone ball launch monitor 200, the parameters that areprocessed and wirelessly transmitted to the mixed reality glasses 100can include the following: Vertical Launch Angle, Horizontal LaunchAngle, Vertical Descent Angle, Smash Factor, Spin Rate, Spin Axis, CarryDistance, Roll Distance, Total Distance, Lateral Landing, Apex Height,Flight Time, Shot Dispersion, Shot Distance to Pin, Skills Score, Lowpoint, Club Speed, Club Speed Profile, Club Acceleration Profile, Faceto Path, Face to Target, Dynamic Loft, Angle of Attack, Club Path, SpinLoft, Swing Plane Horizontal, and Swing Plane Vertical.

With an electronic ball 300, the parameters that are processed andwirelessly transmitted to the mixed reality glasses can include thefollowing: GPS Location, Distance of Roll, Spin Rate, Distance to Pin,Time and Distance of Carry, GPS Point of Carry End, Ball Speed, Angle ofDescent, Carry Flight Path, Draw/Fade/Altitude High Delta, Carry HeightApex.

The golf ball's flight path can be represented within the system by apolynomial, using known techniques. This polynomial can be evaluated andprocessed by a processor in the mixed reality glasses 100 (e.g.,graphics and application processor 103) until the simulated ball pathintersects a polygon mesh representing the ground. Standard rigid bodyphysics calculations can then be performed to simulate the ball trackalong the ground polygon mesh.

The system may include software algorithms and/or hardwired logic thatcan use gyroscope/accelerometer/compass data to determine head rotationand a gravity sensor to determine the down direction. This data can beprocessed with, for example, a sensor fusion Kalman filter function orusing functions of the Unity graphical software develop tool. The ball'sflight path is then processed to generate various data that is displayedto the user, such as ball velocity, launch angle, ball spin, etc. Thesefunctions can be implemented, at least in part, by the graphics andapplication processor 103 of the mixed reality glasses 100.

The mixed reality glasses 100 provide both visual data and visualenvironment feedback to the user. The data feedback can include golfsimulation data, such as statistics on the shot itself, such as ballspeed, distance traveled, spin rate, spin orientation, golf club speed,and club orientation. The visual environment feedback can includedisplaying a fully immersive virtual environment to the user to allowthe user to see the golf course or driving range (or any other type ofplaying field, e.g., if the system is implementing a virtual sport otherthan golf) as if the user were actually there and/or as virtual dataoverlaid on the user's real world view. The virtual environment mayinclude computer generated models or computer enhanced photographicrepresentations of a real golf course. Hence, the user's view of avirtual golf course or driving range (or other type of playing field,court, etc.) through the mixed reality glasses 100 can be based on a 3Drendered version of a golf course or driving range, or digital photos ofa real golf course or driving range, or a combination of 3D renderingand digital photographic data. Alternatively, the ball motion andtracking data can be overlaid on the user's real world view.

One or more 360° panoramic photos of a real golf course (or multiplegolf courses) can be digitally encoded and stored in memory in the mixedreality glasses 100, e.g., in equirectangular format, and rendered ontoa graphical sphere for display. The user's viewpoint (i.e., a virtualcamera) can be computationally located at the center of this virtualsphere. The 360° panoramic photos can be identified by GPS coordinatesand can be loaded according to the physical ball location in the realworld.

The system enables the user to hit an actual golf ball within a limitedphysical space, or on a full course or driving range, and obtainrealistic visual feedback of what the golf ball trajectory would be on areal golf course. The visual feedback is displayed to the user by themixed reality glasses 100. The visual feedback, which can include golfball data and club data, and the virtual environment display can be usedas an entertainment device. The user can virtually play a round of golf,or hit multiple shots at a particular hole or practice driving in avirtual driving range, or the user can play an actual golf course orpractice at an actual driving range and use the system to virtuallydisplay the ball motion and tracking data over the real world view. Thedisplay can also include additional elements, such as game play elementsfor entertainment value.

In some embodiments and/or operational modes, the displayed virtualimages (e.g., images of a virtual golf course) will appear to the useronly when the user looks in a specific direction, such as the directionthat the user has calibrated the mixed reality glasses to correspond tothe direction of the hole. When the user looks in any other direction,the mixed reality images will not be displayed (except that certain ARdata may be displayed near the edge of the user's field of view), suchthat his view of the real world will be mostly or completelyunobstructed. Hence, when the user is looking down at the ball gettingready to swing, the user sees an unobstructed view of the ball and thesurrounding ground; but as soon as the user looks up in the direction ofthe hole (i.e., the “target area”), the mixed reality images of the golfcourse will appear instantly, or nearly instantly, overlaid on or inplace of the user's view of the real world. In at least someembodiments, it may be desirable to provide a user-perceptibletransition (e.g., fade-in/out) from a non-display state to display state(and vice versa) of those images, to improve the user's visual comfort,since an instantaneous visual transition may be disconcerting to theuser.

FIG. 8 illustrates an example of the user's view through the mixedreality glasses 100 of a regulation golf ball when looking down inpreparation to hit the ball, showing an unobstructed real world view ofthe ball. In contrast, FIG. 9 illustrates a corresponding example of theuser's view through the mixed reality glasses 100 when the user looks upafter hitting the ball, including the panoramic view of a virtual golfcourse and head-up display (HUD) elements 901-908 that include golfsimulation data. In this example view, the displayed images completelyblock the user's view of the real world. In other instances, however, itmay be desirable for the displayed images to cover only part of theuser's field of view and/or to make at least some of the displayedimages partially transparent.

The system also enables multiple users to connect in a common virtualenvironment. Users separated by distance, but operating the system atthe same time, can thereby play against each other, or as a team againstother teams, in a virtual golf contest. Each user will see the virtualenvironment from their own individual perspective. Moving avatars of theother players can be displayed to each user. Avatars may becustomizable, standardized or may represent popular professionalgolfers. Avatars movements may closely replicate the general orientationof the user, especially the orientation and movement of the user's headby using the mixed reality glasses sensors.

A user operates and controls the system though a user interface that caninclude physical buttons (not shown) on the mixed reality glasses 100and/or alternative input devices, such as hand gesture and/or voicecommand recognition. The user input device(s) may be on the mixedreality glasses 100 and/or in a separate hand-held device (not shown)that communicates wired or wirelessly with the system, such as theuser's smartphone, watch or local computer system.

In at least some embodiments, the system provides several modes ofoperation, including a Virtual Range mode, a Virtual Caddy mode, aVirtual Course mode and a Virtual Instructor mode. Each of these modesof operation supports the use of mixed reality glasses 100 to providevisual feedback of the ball tracking data and simulation graphics.

The Virtual Range mode supports simulated graphics and ball trackingdata superimposed on the real-world view while using a real golf drivingrange or while hitting into a net. The simulated graphics may include avariety of animated targets and other entertainment style game playoptions. FIG. 10 illustrates an example of the user's view through themixed reality glasses while using the Virtual Range mode. The viewincludes virtual graphical data, such as simulated targets 1001 and HUDdata 1002, within a completely virtualized golf driving range orsuperimposed over the user's real world view of a real driving range, toenhance golf play at a driving range or other location.

In one embodiment of the Virtual Range mode the user experience isenhanced when hitting golf balls at a real driving range. The useroperates the system by first turning on the mixed reality glasses 100,then places a golf ball on a tee that will be tracked using an externallaunch monitor or an electronic ball connected, by a wireless link, tothe glasses. The user can calibrate, i.e., center, the mixed realitydisplay showing a mixed reality view overlaid on the real world view.The view can be calibrated by looking in the direction in which the ballis intended to be hit and selecting the calibration function, e.g., byusing either a switch, voice commands, or head/eye movement input. Oncethe view is calibrated the displayed mixed reality images will lock tothat virtual position and will stay in that position regardless of wherethe user looks.

The mixed reality view includes various graphical elements that enhancethe user experience. These graphical elements may be three-dimensional(3D), animated graphics showing a variety of targets that the user canaim at while hitting the ball at the range. After the user hits the balland looks up in the direction of the ball, the display will show agraphical ball track along with animated graphics that react to andchange based on the ball position along with scoring information. Thedisplay may also show various AR images/data, such as ball speed, shotdistance, etc.

The Virtual Range mode also allows for multiple players in a group.Other users connected as a group will also be able to see the user'sshot data along with current score. The user will be able to see shotand score data of the other players connected in the group.

The Virtual Caddy mode provides the user with ball tracking and locationdata while golfing on a real golf course. FIG. 11 shows an example ofthe user's view through the mixed reality glasses while in the VirtualCaddy mode, with the HUD elements 1101-1108 displaying ball trackingdata superimposed over the user's real world view of a golf coursevisible when the user looks up after hitting the ball.

In one embodiment, the user operates the system by first turning on themixed reality glasses 100 of the Virtual Caddy mode, then places on atee an electronic ball that is electronically connected, by a wirelesslink, to the glasses. The user can calibrate, i.e. center, the mixedreality HUD by looking in the direction in which the ball is intended tobe hit and selecting the calibration function, e.g., by using either aswitch, voice commands, or head/eye movement input. Once the HUD iscalibrated the display will lock to that virtual position and will stayin that position regardless of where the user looks. The real world viewis fully visible in all view directions, with the HUD data overlaid onthe real world view. After the user hits the ball and looks up in thedirection of the ball, the HUD display will show a graphical ball trackalong with AR images/data that can include, for example, ball speed,club speed, ball distance, carry distance launch angle, ball spin,flight time and score. After the ball lands the user may initiate a balllocation finder function that will show a HUD target, arrows and audiocues to help the user locate the ball in the real world. In someembodiments, users connected as a group can see the user's shot dataalong with current score, and the user can see shot and score data ofthe other players connected in the group.

The Virtual Course mode allows the user to play a simulated golf course,complete with 360° panoramic views of a virtual golf course (FIG. 9) andincludes a simulation and display of the ball tracking and locationdata. In one embodiment of the Virtual Course mode, the user can hit theball into a net or at a driving range. The user operates the system byfirst turning on the mixed reality glasses 100, then places a regulationball on a tee that will be tracked using an external launch monitorconnected, by a wireless link, to the glasses. The user can calibrate,i.e., center, the mixed reality display showing a virtual view of aspecific golf course hole that the user has selected. The view can becalibrated by looking in the direction in which the ball is intended tobe hit and selecting the calibration function, e.g., by using either aswitch, voice commands, or head/eye movement input. Once the view iscalibrated the display will lock to that virtual position and will stayin that position regardless of where the user looks. The real world viewis visible when the user looks down at the ball, and the virtual courseview is visible when the user looks up and is visible throughout a full360-degree view. After the user hits the ball and looks up in thedirection in which the ball was hit, the display will show a graphicalball track along with data that can include, for example, ball speed,club speed, ball distance, carry distance launch angle, ball spin,flight time and score, overlaid on the virtual course view. After theball “lands” (as computed/predicted by the system), the user may selecta function that virtually transports them to the new ball location. Theuser then places another regulation ball on the tee and hits a shot thatis from that virtual location.

The Virtual Course mode also allows for multiplayers in a group. Otherusers connected as a group will also be able to see the user's shot dataalong with current score. The user will be able to see shot and scoredata of the other players connected in the group. The mixed realityglasses may have microphones and speakers to allow the user to converselive with other players.

The Virtual Course mode may also allow the user to play against one ormore virtual players. For example, while playing in the Virtual Coursemode, the user can have the ability to play against a virtual TigerWoods or other professional golfer. The virtual player avatar will bevisible along with his/her shots and score. The user can play along withthis virtual player though a full game of golf. Other game options maybe a hitting distance contest at a virtual driving range or othermini-games, such as a putting challenge or a hitting over the waterchallenge (e.g., at the island green, hole #17 at Sawgrass). The systemwill allow the user to virtually move about the scene in threedimensions.

The Virtual Instructor mode provides training feedback in the form of,for example, a moving 3D avatar visible to the user in the virtualenvironment, which is essentially a virtual shadow user. Normally theVirtual Instructor mode will operate within the context of one of theother three modes, i.e., Virtual Range, Virtual Caddy or Virtual Course.For example, the user can invoke the Virtual Instructor while playing around of golf in the Virtual Course mode. When operating in the VirtualInstructor mode, a virtual instructor provides feedback to the user,based on the shot data, in any of various forms, such as visual motionfeedback, video replay of the user's avatar, video playback of storedimages for instruction and audio feedback (e.g., recorded or synthesizedspeech) for short explanations and suggestions. The virtual instructormay respond to the user's questions posed verbally or through anotheruser input device.

FIG. 13 is a drawing showing the user's view through the mixed realityglasses of such a 3D (shadow user) avatar 1300 displayed during theVirtual Instructor mode. The shadow user avatar 1300 allows the systemto visually demonstrate proper setup and form for a particular golfshot. The shadow user avatar 1300 can include highlighted and/or coloredregions indicating key areas of concentration that should be followed inorder to replicate the proper form for a particular golf shot. TheVirtual Instructor mode may use the shadow user avatar 1300 to provideinstruction and feedback.

The system configuration for Virtual Instructor mode, an example ofwhich is shown in FIG. 12, can include several components used todetermine the user's body movement and technique. In this mode there maybe additional sensors attached to the user's body, and/or a video camerato record the user's movement, both of which may be used to determinethe user's motion and technique. The user's body movement and techniquecan be compared to ideal standards, and feedback can be provided to theuser, through audio, video and/or computer-generated graphics, showingmovements or techniques that can be improved.

In Virtual Instructor mode, the user may attach various sensors to theirequipment and/or body. These sensors may include a golf club sensor 601that measure the golf club position and acceleration, head band sensor602 that measures the user's head position and angular/linearacceleration, shoulder sensors 603 that measure the user's shoulderposition and movements, waist sensors 604 that measure the user's waistmovements, leg sensors 605 that measure the user's leg position andmovement, and a pressure sensitive mat or sole inserts worn within theshoes with sensors 606 that measure the user's individual footpressures.

The various sensors, such as the foot pressure sensor, are related tothe user's weight shift throughout the golf swing, which is a key aspectto proper technique resulting in optimal golf swings and ball distance.The virtual instructor software gathers the raw sensor information,analyzes it to determine the user's overall motion and technique andcompares this to an ideal motion. Any discrepancies between the idealmotion and the user's motion can be displayed to the user along withinstructions, either audio, video or computer-generated graphics, ontechniques that may be employed to help the user practice motions andpositions that better match the ideal.

The Virtual Instructor mode may also include a standard, stereo oradvanced technology camera 607 that is used to motion capture the user'smovements. This motion capture, in addition to or in place of the bodysensors, may be used to determine the user's movements, both overall andfor key locations on the user's body. This data is compared to the idealmovement and feedback is provided to the user, using audio, video orcomputer-generated graphics, along with instructional information thatthe user may use to practice improving their movements and overalltechnique.

As noted above, the system's mixed reality glasses 100 can include aplurality of sensors that include, but are not limited to, positional,rotational and acceleration feedback. This feedback may be used the bygolf simulation software to determine the position, orientation andspeed of the user's head before, during and after the shot. This datamay be used in the training portion of the simulation to monitor forproper head orientation and stability and to provide both visual andaudio feedback of proper and improper movements.

There are several advantages to the system over the existing technology.The mixed reality glasses 100 provide a small, portable visual feedbacksystem that is not available with systems utilizing projection screensor larger computer monitors. The mixed reality glasses 100 provide avirtual environment that is properly oriented to the user's field ofview and orientation, which is not available with existing systems usinglarge or smaller portable displays. The mixed reality glasses 100provide sensor feedback that can be used to properly train the user tokeep their head still and to keep looking at the ball during the shot,which is not available with existing systems.

The Virtual Instructor mode provides instructional and audio and/orvideo feedback based on the result of hitting the ball. The feedback isbased on the ball tracking data and may also include additional datasensors attached to the user's body to provide further data points forthe instructional algorithms. Virtual Instructor can be used with a netor at a driving range or golf course. A video recording/sharingfunctionality can also be used.

In one embodiment of the Virtual Instructor mode, the user can hit theball into a net, at a driving range or on a golf course. The user's bodyis tracked using sensors that are attached to specific locations on theuser's body, such as the head, shoulders, waist, legs and feet and/orthe golf club. Additionally, or alternatively, tracking the users bodymovement may be done by using a video camera.

The user operates the system by first turning on the mixed realityglasses 100, then places a regulation ball on a tee that will be trackedusing an external launch monitor or an electronic ball connected, by awireless link, to the glasses. The user can calibrate, i.e., center, themixed reality display showing a virtual view of a specific golf coursehole that the user has selected or only a graphical HUD overlaid on thereal world view. The view can be calibrated by looking in the directionof where the ball is intended to be hit and selecting the calibrationfunction, e.g., by using either a switch, voice commands, or head/eyemovement input. Once the view is calibrated the display will lock tothat virtual position and will stay in that position regardless of wherethe user looks. When using a Virtual Course view mode, the real worldview is visible when the user looks down at the ball and the virtualcourse view is visible when the user looks up and is visible throughouta full 360-degree view. After the user hits the ball and looks up in thedirection of the ball, the display will show a graphical ball trackalong with data that can include, for example, ball speed, club speed,ball distance, carry distance launch angle, ball spin, flight time andscore overlaid on the virtual course view or the real world viewdepending on the mode the user has selected.

After the ball is hit a virtual three dimensional avatar representingthe user is displayed in the mixed reality view. This avatar replays theuser's golf swing and body movement, and the user is able to view thisavatar mixed reality view from any angle. The Virtual Instructor logicanalyzes the user's swing and body movement and highlightsless-than-optimal body and swing movements on the user's avatar. Thevirtual instructor describes the movements that were less than optimal,provides instruction on techniques to improve and shows an updatedavatar playing back the optimal swing and body movements for the user tosee the optimal form.

The machine-implemented operations described above can be implemented byprogrammable circuitry programmed/configured by software, or entirely byspecial-purpose circuitry, or by a combination of such forms. Suchspecial-purpose circuitry (if any) can be in the form of, for example,one or more application-specific integrated circuits (ASICs),programmable logic devices (PLDs), field-programmable gate arrays(FPGAs), system-on-a-chip systems (SOCs), etc.

Software to implement the techniques introduced here may be stored on amachine-readable storage medium and may be executed by one or moregeneral-purpose or special-purpose programmable microprocessors. A“machine-readable medium,” as the term is used herein, includes anymechanism that can store information in a form accessible by a machine(a machine may be, for example, a computer, network device, cellularphone, personal digital assistant (PDA), manufacturing tool, any devicewith one or more processors, etc.). For example, a machine-accessiblemedium includes recordable/non-recordable media (e.g., read-only memory(ROM); random access memory (RAM); magnetic disk storage media; opticalstorage media; flash memory devices; etc.), etc.

The term “logic”, as used herein, means: a) special-purpose hardwiredcircuitry, such as one or more application-specific integrated circuits(ASICs), programmable logic devices (PLDs), field programmable gatearrays (FPGAs), or other similar device(s); b) programmable circuitryprogrammed with software, such as one or more programmed general-purposemicroprocessors, digital signal processors (DSPs) and/ormicrocontrollers, system-on-a-chip systems (SOCs), or other similardevice(s); or c) a combination of the forms mentioned in a) and b).

Any or all of the features and functions described above can be combinedwith each other, except to the extent it may be otherwise stated aboveor to the extent that any such embodiments may be incompatible by virtueof their function or structure, as will be apparent to persons ofordinary skill in the art. Unless contrary to physical possibility, itis envisioned that (i) the methods/steps described herein may beperformed in any sequence and/or in any combination, and that (ii) thecomponents of respective embodiments may be combined in any manner.

Although the subject matter has been described in language specific tostructural features and/or acts, it is to be understood that the subjectmatter defined in the appended claims is not necessarily limited to thespecific features or acts described above. Rather, the specific featuresand acts described above are disclosed as examples of implementing theclaims and other equivalent features and acts are intended to be withinthe scope of the claims.

The invention claimed is:
 1. A mixed-reality golf simulation systemcomprising: a near-eye display including a partially-transparent displayelement configured to display virtual objects over physical objectswithin a user field of view to provide a mixed-reality environment,wherein the near-eye display includes a user input device, wherein thenear-eye display includes one or more head orientation sensors todetermine a head orientation of the user; and a controllercommunicatively coupled to the near-eye display and configured toreceive ball-tracking data from a ball-tracking sub-system, thecontroller including one or more processors configured to executeprogram instructions causing the one or more processors to: direct thenear-eye display to display, based on a head orientation of the userreceived from the near-eye display, virtual objects of a mixed-realityenvironment within at least a portion of the user field of view when thenear-eye display indicates that the user is looking in the targetdirection, wherein the near-eye display provides an at least partiallyunobstructed real-world view through the partially-transparent displayelement when the near-eye display indicates that the user is looking ina direction different than the target direction; receive ball-trackingdata of a golf ball hit by the user in real-time from the ball-trackingsub-system; and direct the near-eye display to display a virtual objectrepresenting a trajectory of the golf ball within the mixed-realityenvironment in real-time when the near-eye display indicates that theuser is looking in the target direction, wherein the trajectory of thegolf ball is based on the ball-tracking data.
 2. The mixed-reality golfsimulation system of claim 1, wherein the mixed-reality environmentincludes at least one of at least one of a fairway, a green, a trap, ora pin.
 3. The mixed-reality golf simulation system of claim 1, whereinthe at least partially unobstructed real-world view includes a fullyunobstructed real-world view.
 4. The mixed-reality golf simulationsystem of claim 1, wherein at least one of a location or an orientationof the user in the mixed reality environment is selectable with the userinput device.
 5. The mixed-reality golf simulation system of claim 1,wherein the one or more virtual objects of the mixed-reality environmentinclude one or more augmented reality objects displayed over a portionof the field of view, wherein at least a portion of the field of view isunobstructed to provide visualization of physical objects through thepartially-transparent display element, wherein a displayed perspectiveof the one or more augmented reality objects are continually updatedbased on at least one of a head orientation of the user from thenear-eye display or a location of the user within the mixed realityenvironment.
 6. The mixed-reality golf simulation system of claim 1,wherein the one or more virtual objects of the mixed-reality environmentinclude a virtual reality scene obstructing at least a portion of thefield of view from a vantage point of the user at a selected locationwithin the virtual reality scene, wherein a displayed perspective of thevirtual reality scene is continually updated based on at least one of ahead orientation of the user received from the near-eye display or alocation of the user within the mixed reality environment.
 7. Themixed-reality golf simulation system of claim 1, wherein the near-eyedisplay is configured to the one or more virtual objects of the mixedreality environment between a display state and a non-display statebased on a head orientation of the user from the near-eye display. 8.The mixed-reality golf simulation system of claim 1, wherein thetrajectory of the golf ball hit by the user is limited by a distancelimiting device, wherein the virtual object representing a trajectory ofthe golf ball within the mixed-reality environment in real-time includesa predicted trajectory based on the ball-tracking data over a launchwindow ending when motion of the golf ball is hindered by the distancelimiting device.
 9. The mixed-reality golf simulation system of claim 8,wherein an opacity of the partially-transparent display element isadjustable.
 10. The mixed-reality golf simulation system of claim 1,wherein the virtual object representing the trajectory of the golf ballcomprises: a persistent trail indicative of at least a portion of thetrajectory.
 11. The mixed-reality golf simulation system of claim 1,wherein the one or more processors are further configured to executeprogram instructions causing the one or more processors to: direct thenear-eye display to display virtual objects representing at least one ofa launch velocity, a launch angle, a hook angle, a rotation axis, arotation rate, a hang time, a carry distance, a roll distance, or atotal distance of the golf ball.
 12. The mixed-reality golf simulationsystem of claim 1, wherein the mixed-reality environment comprises: anavatar associated with a remote user.
 13. The mixed-reality golfsimulation system of claim 1, wherein the one or more processors arefurther configured to execute program instructions causing the one ormore processors to: direct the near-eye display to provide coachingfeedback to the user based on the trajectory, wherein the coachingfeedback includes at least one of audio information or virtual objectsdisplayed over at least a portion of the user field of view.
 14. Themixed-reality golf simulation system of claim 1, further comprising: auser-tracking sub-system including one or more sensors to generateuser-motion data indicative of motion of the user during a shot.
 15. Themixed-reality golf simulation system of claim 14, wherein the one ormore sensors comprise: at least one of a camera or one or more wearablesensors.
 16. The mixed-reality golf simulation system of claim 15,wherein the one or more processors are further configured to executeprogram instructions causing the one or more processors to: direct thenear-eye display to display a virtual object including an avatarrepresenting the user after the user hits the golf ball, wherein theavatar simulates motion of the user hitting the golf ball based on theuser-motion data, wherein the avatar is viewable from any angle in themixed-reality environment.
 17. The mixed-reality golf simulation systemof claim 16, wherein the ball-tracking sub-system comprises: at leastone of a camera, a doppler-tracking device, a LIDAR-tracking device, ora sensor located within the golf ball.
 18. A mixed-reality golfsimulation system comprising: a near-eye display including apartially-transparent display element configured to display virtualobjects over physical objects within a user field of view to provide amixed-reality environment, wherein the near-eye display includes a userinput device, wherein the near-eye display includes one or more headorientation sensors to determine a head orientation of the user; and acontroller communicatively coupled to the near-eye display andconfigured to receive ball-tracking data from a ball-trackingsub-system, the controller including one or more processors configuredto execute program instructions causing the one or more processors to:direct the near-eye display to display, based on a head orientation ofthe user received from the near-eye display, virtual objects of amixed-reality environment within at least a portion of the user field ofview; direct the near-eye display to provide, based on the headorientation of the user received from the near-eye display, anunobstructed real-world view through the partially-transparent displayelement of the location from which the golf ball is to be hit; receiveball-tracking data of a golf ball over the launch window in real-timefrom the ball-tracking sub-system as the user hits the golf ball; anddirect the near-eye display to display a virtual object representing thegolf ball moving along a predicted trajectory after the launch windowwithin the mixed-reality environment, wherein the predicted trajectoryis determined based on the ball-tracking data within the launch window.19. The mixed-reality golf simulation system of claim 18, wherein thelaunch window ends when motion of the golf ball is hindered by thedistance limiting device, wherein the distance limiting device includesat least one of a net or a tether.
 20. A multi-user mixed-reality golfsimulation system comprising: two or more near-eye displays, eachnear-eye display including a partially-transparent display element to beworn by at least one of the two or more users, wherein the two or morenear-eye displays are configured to display virtual objects overphysical objects within user fields of view to provide a mixed-realityenvironment, wherein each of the two or more near-eye displays include auser input device, wherein each of the two or more near-eye displaysinclude a head orientation sensor to determine head orientations of thetwo or more users; and a controller communicatively configured to thetwo or more near-eye displays and configured to receive ball-trackingdata from a ball-tracking sub-system, the controller including one ormore processors configured to execute program instructions causing theone or more processors to: direct the two or more near-eye displays todisplay, based on head orientations of the two or more users receivedfrom the two or more near-eye displays, virtual objects of a commonmixed-reality environment from different locations associated withlocations of the two or more users, wherein a near-eye display of thetwo or more near-eye displays worn by a user of the two or more usershitting a golf ball of the one or more golf balls provides anunobstructed real-world view through the partially-transparent displayelements of a location from which the golf ball is to be hit when thenear-eye display worn by the user hitting the golf ball indicates thatthe user hitting the golf ball is looking in the direction from whichthe golf ball is to be hit, wherein the location from which the golfball is to be hit is selectable with a user input device on the near-eyedisplay worn by the user hitting the golf ball; receive ball-trackingdata of one or more golf balls hit by any of the two or more users inreal-time from the at least one ball-tracking sub-system; and direct thetwo or more near-eye displays to display virtual objects representingtrajectories of the one or more golf balls hit by at least one of thetwo or more users in real-time, wherein the trajectories are determinedbased on the ball-tracking data, wherein the trajectories are displayedon the two or more near-eye displays based on the locations of the twoor more users within the mixed-reality environment.